Measuring cervical spine posture using nostril tracking

ABSTRACT

A method for detecting deviation from a preferred cervical spine posture when using a mobile device is disclosed. The mobile device uses a front-facing camera to capture images of the user and apply a nostril tracking algorithm to the images. The nostril tracking algorithm is used in real-time to measure displacement of the user&#39;s nostrils and correlate the nostril displacement to a cervical spine flexion angle. The user&#39;s cervical spine flexion angle is communicated using an alarm device, such as a row of lights, which allows the user to monitor and correct their posture and avoid potential injury.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.62/191,574, filed Jul. 13, 2015.

FIELD

The present disclosure relates generally to a computing devices used toimprove posture. More particularly, the disclosure relates to using afront-facing camera of a mobile device to monitor cervical spineposture.

BACKGROUND

The use of computing devices, including desktop computer, laptops,tablets, and mobile phones, has increased dramatically over the last fewdecades. Many occupations require some form of computer work andsometimes the same individual will continue to use a computing device athome for leisure. Without proper ergonomic assessments or interventions,the users of these computing devices may be subjecting themselves topoor posture and potential musculoskeletal injuries.

There has been substantial research and guidelines surrounding the useof traditional desktop computers but much of this research does notapply to mobile computing devices. Much of the current researchinvestigating computer and mobile device usage have used sittingpostures, short task durations including breaks, and have not observedthe frequency of postural changes. This research is not representativeof the frequency and duration of mobile device usage from modern dayusers.

Recent research has discovered that prolonged use of mobile devices canlead to poor posture, particularly a forward-tilted head posture thatputs increased stress on cervical vertebrae. This poor posture can becaused by users hunching over and flexing their neck forward when usingtheir mobile device for prolonged periods. The weight seen by the spinedramatically increases when flexing the head forward at varying degrees.Loss of the natural curve of the cervical spine leads to incrementallyincreased stresses about the cervical spine. These stresses may lead toearly wear, tear, degeneration, and possibly surgeries. Mobile devicesare also more frequently used by teenagers and youths who are stilldeveloping their musculoskeletal system. Forward-tilted head posture hasalso been shown to promote changes in heart rate and breathing.

There have been several human head and face tracking solutions formonitoring or improving posture when using a mobile device. All of thesesolutions have various problems. Some of them cannot work in real-time,others are not robust enough. Some methods even need special equipmentin their application, such as an IR camera to locate and track pupils byhighlighting them in images.

Liu, J., Liu, C. & Zhao, Z. (2012). “Head gesture recognition based onLK algorithm and GentleBost”. Advances in Information Sciences andService Sciences, (4), 4, 158-167 (hereinafter referred to as Liu etal.) describes the use of nostril tracking to monitor human headmovements such as nodding, shaking, bowing, etc. Nostril tracking isfound to be robust as nostril shape does not change with age, race orgender.

SUMMARY

According to a first aspect, a method is for detecting deviation from apreferred cervical spine posture when using a mobile device. The methodcomprises obtaining a calibration image from a front-facing camera ofthe mobile device; locating nostrils in the calibration image; obtaininga plurality of tracking images obtained from the front-facing camera ofthe mobile device and locating the nostrils in the tracking images;measuring a nostril displacement of the nostrils between the trackingimages and the calibration image; correlating the nostril displacementto a cervical spine flexion angle; and comparing the cervical spineflexion angle to an acceptable range of cervical spine flexion.

In some aspects, the method can also associate the difference betweencervical spine flexion angle and the acceptable range of cervical spineflexion to an alarm severity and indicate the alarm severity using analarm device coupled to the mobile device. The alarm severity can beincreased according to a length of time that the cervical spine flexionangle is outside the acceptable range of cervical spine flexion.Obtaining the calibration image can also include providing instructionsfor the user to position their head and neck in a neutral position. Themethod can use a Lucas-Kanade method using a least squares criterion tomeasure the nostril displacement between tracking images.

In some aspects, locating the nostrils can include performing facialdetection to define a face on any one of the calibration image or atleast one of the tracking images; and obtaining a nostril region portionof the image. The nostril region portion of the image can be 2/7 to 5/7of the face width and ½ to ¾ the face height. In a further aspect,locating the nostrils can include identifying one or more nostrilcandidates in the image, the nostril candidate having a small pixelwindow relative to the image size; applying an edge detection algorithmon the nostril candidates to obtain a nostril candidate matrix for eachnostril candidate; and comparing each nostril candidate matrix to anostril classifier matrix to identify the nostril. The edge detectionalgorithm can use a Gabor filter. The nostril candidate matrix can beobtained by varying the Gabor filter coefficients, filter orientation,and spatial frequency of the Gabor filter. The nostril region portion ofthe image can be gray-scaled to assist with edge detection.

In still other aspects, the nostril classifier matrix can be obtainedusing a machine learning boosting algorithm trained against arepresentative sample of positive nostril candidates and negativenostril candidates. The machine learning boosting algorithm can beGentle Boost algorithm variant. Comparison of the nostril candidatematrices with the nostril classifier matrix can seek to minimize theweighted error with the nostril classifier matrix.

According to a second aspect, a mobile device is disclosed comprising aforward facing camera; a display; a memory for storing instructions; andone or more processors configured to execute the instructions to: obtaina calibration image from the front-facing camera; locate nostrils in thecalibration image; obtain a plurality of tracking images obtained fromthe front-facing camera and locate the nostrils in the tracking images;measure a nostril displacement of the nostrils between the trackingimages and the calibration image; correlate the nostril displacement toa cervical spine flexion angle; and compare the cervical spine flexionangle to an acceptable range of cervical spine flexion.

In some aspects, the mobile device can further include an alarm device,and the one or more processors can be further configured to associatethe difference between cervical spine flexion angle and the acceptablerange of cervical spine flexion to an alarm severity; and indicate thealarm severity to the alarm device. The alarm device can be a row oflights or it can be integrated with the mobile device display. It canalso provide a progressive scale to indicate severity of deviation fromthe preferred posture.

In some aspects, the processors can be further configured to identifyone or more nostril candidates in the image, the nostril candidatehaving a small pixel window relative to the image size; apply an edgedetection algorithm on the nostril candidates to obtain a nostrilcandidate matrix for each nostril candidate; and compare each nostrilcandidate matrix to a nostril classifier matrix to identify the nostril.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the various embodiments described hereinand to show more clearly how they may be carried into effect, referencewill now be made, by way of example only, to the accompanying drawingswhich show at least one exemplary embodiment, and in which:

FIG. 1 is a block diagram of a mobile device having a front-facingcamera;

FIG. 2. is a flowchart diagram of a method for detecting deviation froma preferred cervical spine posture when using the mobile device of FIG.1;

FIG. 3 is a flowchart diagram of a method for locating nostrils in animage that can be used in the method of FIG. 2;

FIG. 4 is a photograph of an individual identifying a face detectedregion and a nostril region within the face detected region.

DESCRIPTION OF VARIOUS EMBODIMENTS

It will be appreciated that for simplicity and clarity of illustration,where considered appropriate, numerous specific details are set forth inorder to provide a thorough understanding of the exemplary embodimentsdescribed herein. However, it will be understood by those of ordinaryskill in the art that the embodiments described herein may be practicedwithout these specific details. In other instances, well-known methods,procedures and components have not been described in detail so as not toobscure the embodiments described herein. Furthermore, this descriptionis not to be considered as limiting the scope of the embodimentsdescribed herein in any way, but rather as merely describing somepossible implementations of various embodiments.

The embodiments of the systems, devices and methods described herein maybe implemented in hardware or software, or a combination of both. Someof the embodiments described herein may be implemented in computerprograms executing on programmable computers, each computer comprisingat least one processor, a computer memory (including volatile andnon-volatile memory), at least one input device, and at least one outputdevice. For example, and without limitation, the programmable computersmay have multiple processors and at least one network interface device.Program code may operate on input data to perform the functionsdescribed herein and generate output data.

Reference is first made to FIG. 1, shown is a block diagram of a mobiledevice 100 that can include a processor 110, memory 120, display 130,and input device 140. A network interface 150 can be provided to allowmobile device 100 to communicate with other computing devices over acommunication network. Mobile device 100 can further include afront-facing camera 160, a microphone 170, and a speaker 180.Front-facing camera 160 is a camera that is positioned to capture animage of the user of mobile device 100. Front-facing camera 160 has asufficient resolution to capture and measure nostril displacement, andis preferably capable of capturing images at high-definitionresolutions. Examples of mobile device 100 can include mobile computingdevices, such as mobile phones, laptop computers or tablet computingdevices.

An alarm device 190 can be included on mobile device 100 to alert a userto improper posture. Alarm device 190 can be a progressive alarm thatindicates the severity of the deviation from a preferred posture. Forexample, alarm device 190 can include a row of lights (e.g. a row oflight emitting diodes across the top of the display) with each lightrepresenting an increasing severity of posture deviation from left toright. Some embodiments could also use a color scale to representseverity, for example, by using green, amber and red lights. Otherembodiments could use a row of lights to indicate proper posture, forexample, the more lights that are maintained, then the more correct thecervical posture and less harmful to the user so that the user isencouraged to maintain all of the lights to have a proper posture.

In some embodiments, alarm device 190 can be implemented in software andintegrated with display 130 such that a portion of the display 130 isused to indicate the severity of posture deviation. For example, a topportion of the display can be used to display a row of lights, a bar, ora changing color light that indicate severity of posture deviation.Other embodiments of mobile device 100 can also use alarm device 190 tocontrol settings (such as brightness, contrast, color settings, etc.) ofdisplay 130 such that display 130. For example, if a significant posturedeviation is detected then alarm device 190 can change the settings ofdisplay 130, for example by dimming the brightness of the display, untilthe user corrects their posture.

Referring now to FIG. 2, shown is a method 200 for detecting a deviationfrom a preferred cervical spine posture. Method 200 can be implementedby program instructions stored in memory 120 and executed by processor110 of mobile device 100. Method 200 uses a nostril detection anddisplacement algorithm to measure deviation from a neutral cervicalspine position quickly and to provide feedback to using alarm device190.

At step 202 a calibration image is obtained from a front-facing camera160 of mobile device 100. Step 202 can be initiated by a user input tobegin the calibration process. A demonstrational video or diagrams canbe shown on display 130 to illustrate a preferred neutral spineposition. The preferred cervical spine posture is a neutral positionwith the individuals ear lobes aligned above the shoulders. In someembodiments, step 202 can further include capturing calibration imageswith the head and neck in various positions in order to provide otherreference points for measuring nostril displacement.

The calibration process can further include ambient light detection anduse of a flash associated with the camera to obtain suitable images.Preferably, the user is instructed to operate in an environment withadequate lighting. Some embodiments could also employ infrared imagingusing an additional sensor to assist with darker environments.

Next at step 204, nostrils on the user's face are located within thecalibration images. Nostril location can be performed using an imagerecognition algorithm such as that detailed in FIG. 3. Foregroundextraction can also be involved in the calibration process to assistwith locating nostrils in the image. Calibration will be successful ifnostrils are located in the calibration image and success can beindicated on alarm device 190, for example, by lighting up the entirerow of lights or a status bar on display 130. If calibration isunsuccessful, the user will be directed to repeat step 202 and correctthe error (e.g. poor lighting, face outside of frame, etc.).

After calibration steps, a series of tracking images are continuallycaptured by front-facing camera 160 and the nostrils of the user arelocated in the tracking images in step 206. Head movement is acontinuous and must be measured over a series of tracking images.Multiple tracking images are also required to discard spurious headmovements that may be unrelated to posture (e.g. turning away from thecamera, nodding, etc.).

Tracking images can be captured using real-time video (e.g. at a highcapture rate, such as 60 images per second) or at a more delayed rate(e.g. three images per second). The nostril tracking and displacementmeasuring algorithms described herein, such as those described withrespect to FIG. 3, allow for capturing and processing tracking images ata high rate. In some embodiments the capture rate can vary in order toconserve battery power. An exponential back-off algorithm can be used tovary the capture rate when there is no posture deviation detected.

Next, at step 208, the nostril displacement is measured between thecaptured tracking images and the calibration images. Computer visionalgorithms can be used measure displacement. Preferably, theLucas-Kanade method (or LK method), or a variant thereof, is used thatrelies on the optical flow between images to be essentially constant ina local neighborhood of the feature under consideration, and solves thebasic optical flow equations for all pixels in that neighborhood usingthe least squares criterion. This method allows the displacement betweentracking images (or a series of tracking images) to be calculatedquickly.

Other head movements can be discarded from consideration due to theirtemporal qualities. For example, brief head nods, shakes, bows, turningto face left or turning to face right should not cause variations in thealarm severity or be associated with a poor cervical spine position.

The measured nostril displacement is then correlated to a cervical spineflexion angle at step 210. For example, a ten pixel displacementdownwards in a tracking image from the calibration image can correspondto a 5 degree cervical spine flexion angle. Calibration step 202 canalso capture an additional calibration image with having a certain knowncervical spine flexion angle, for example, by capturing an image withthe user putting their chin on their chest which is equivalent to a 90degree cervical spine flexion angle. Measuring displacement step 208 andcorrelation step 210 can compare displacement between multiplecalibration images having known cervical spine angles to determine acervical spine flexion angle.

The cervical spine flexion angle determined in step 210 is next comparedto an acceptable range of cervical spine flexion at step 212. The bestposture range is between neutral and 10 degrees of cervical spineflexion, but a range between 15 and 25 degrees of cervical spine flexionwould also be considered acceptable. A poor cervical posture can be anycervical spine flexion angle that exceeds 30 degrees.

At step 212, the greater the difference in cervical spine flexion anglefrom the acceptable range of cervical spine flexion can be associatedwith an alarm severity. This alarm severity can then be indicated to byalarm device 190 so that the user of mobile device 100 is aware of anypoor posture and can correct their posture. Preferably, alarm device 190is a progressive alarm device to indicate severity.

Alarm severity calculation in step 212 can also incorporate temporalfactors. For example, if a non-ideal cervical posture is held for a toolong of a time then alarm severity can be increased to encourage theuser to move from a static posture. Alarm severity can also be increasedafter any prolonged period to encourage the user to stretch or performposture exercises. Also, temporary movements, such as a head nod, can bediscarded from affecting alarm severity.

Reference is now made to FIG. 3, a flowchart diagram of a nostrillocating method 300 for locating nostrils in an image that can be usedin the method 200 of FIG. 2. Nostril locating method 300 uses a machinelearning algorithm to quickly ascertain whether a potential nostrilcandidate is in fact a human nostril. Nostril locating method 300 can beused on calibration images and tracking images.

First, at step 302, a facial detection algorithm is used to identify aface within an image captured by front-facing camera 160. Preferably,facial detection algorithm can operate quickly to define the bounds of aface within the image. Any known facial detection algorithm can be usedwith a preference for facial detection algorithms that can identify thefacial boundary quickly. Using the facial detection outputs, a nostrilregion portion of the image is obtained.

The nostril region portion of the image can be obtained using knownfacial dimensions. For example, using the facial boundary the nostrilregion portion of the between 2/7 to 5/7 of the face width and between ½and ¾ of the face height. The image in FIG. 4 illustrates an image 400where the facial boundary 402 is identified and the nostril regionportion 404 of image 400 is identified.

Returning to FIG. 3, at step 304 nostril candidates are identifiedwithin the nostril region portion 404. Image 400 illustrates a number ofnostril candidates 406 that can potentially be a human nostril in image400. Nostril candidates are focused on a smaller pixel window relativeto the image size (e.g. 3×3 pixels, 5×5 pixels, 7×7 pixels).

Next, in step 306, an edge detection algorithm is applied to eachnostril candidate window to obtain matrix for each nostril candidatewindow by varying the edge detection aspects. Preferably a Gabor filteris used as the edge detection algorithm and the matrix can be obtainedby varying the Gabor filter coefficients, filter orientation, and thespatial frequency of the Gabor filter. It may also be preferable togray-scale each nostril candidate window (or a relevant portion of theimage) prior to applying edge detection algorithms in step 306.

For example, a feature vector can be extracted from a 13×13 pixel imagenostril candidate window centered on a nostril candidate. The nostrilcandidate window can be filtered with a pool of 48 Gabor filters at 8orientations and 6 spatial frequencies (2:12 pixels/cycle at ½ octavesteps) to obtain a nostril candidate matrix of feature vectorsrepresenting that nostril candidate. Fewer Gabor filters, orientationsor spatial frequencies can be used reduce processing requirements formore efficient and faster nostril identification.

The nostril candidate matrix can then be compared to a nostrilclassifier matrix to identify a nostril in step 308. Using a nostrilclassifier matrix allows for a computationally efficient and fast methodof identifying nostrils in the image that can function in real-time on aprocessor in a mobile device. The comparison method can use a weightedleast-squared error in comparing each of the nostril candidate matriceswith the nostril classifier matrix that seeks to minimize the weightederror below a certain detection threshold.

The nostril classifier matrix can be obtained using a machine learningalgorithm that is trained against a representative sample of positivenostril candidates (i.e. nostril candidate window contains a humannostril) and negative nostril candidates (i.e. nostril candidate windowdoes NOT contain a human nostril). The machine learning algorithm can bea boosting algorithm, such as the GentleBoost algorithm. Learningexamples can use a smaller image window (e.g. 5×5 pixel window) centeredon the proposed nostril feature to obtain the nostril classifier matrix.The nostril classifier matrix can contain a number of positively andnegatively weighted vectors obtained from the examples.

While the exemplary embodiments have been described herein, it is to beunderstood that the invention is not limited to the disclosedembodiments. The invention is intended to cover various modificationsand equivalent arrangements included within the spirit and scope of theappended claims, and scope of the claims is to be accorded aninterpretation that encompasses all such modifications and equivalentstructures and functions.

The invention claimed is:
 1. A method for detecting deviation from apreferred cervical spine posture when using a mobile device, the methodcomprising: obtaining a calibration image from a front-facing camera ofthe mobile device; locating nostrils in the calibration image; obtaininga plurality of tracking images obtained from the front-facing camera ofthe mobile device and locating the nostrils in the tracking images;measuring a nostril displacement of the nostrils between the trackingimages and the calibration image; correlating the nostril displacementto a cervical spine flexion angle; and comparing the cervical spineflexion angle to an acceptable range of cervical spine flexion.
 2. Themethod of claim 1 further comprising: associating the difference betweencervical spine flexion angle and the acceptable range of cervical spineflexion to an alarm severity; and indicating the alarm severity using analarm device.
 3. The method of claim 2 further comprising: increasingalarm severity according to a length of time that the cervical spineflexion angle is outside the acceptable range of cervical spine flexion.4. The method of claim 1 wherein obtaining the calibration image furthercomprises: providing instructions for the user to position their headand neck in a neutral position.
 5. The method of claim 1 whereinmeasuring the nostril displacement between tracking images uses aLucas-Kanade method using a least squares criterion.
 6. The method ofclaim 1 wherein locating nostrils further comprises: performing facialdetection to define a face on any one of the calibration image or atleast one of the tracking images; and obtaining a nostril region portionof the image.
 7. The method of claim 6 wherein the nostril regionportion of the image is 2/7 to 5/7 of the face width and ½ to ¾ the faceheight.
 8. The method of claim 1 wherein locating nostrils furthercomprises: identifying one or more nostril candidates in the image, thenostril candidate having a small pixel window relative to the imagesize; applying an edge detection algorithm on the nostril candidates toobtain a nostril candidate matrix for each nostril candidate; andcomparing each nostril candidate matrix to a nostril classifier matrixto identify the nostril.
 9. The method of claim 8 wherein the edgedetection algorithm is a Gabor filter.
 10. The method of claim 9 whereinapplying the edge detection algorithm to obtain the nostril candidatematrix comprises varying the Gabor filter coefficients, filterorientation, and spatial frequency of the Gabor filter.
 11. The methodof claim 8 further comprising gray-scaling the one or more nostrilcandidates.
 12. The method of claim 8 wherein the nostril classifiermatrix is obtained using a machine learning boosting algorithm trainedagainst a representative sample of positive nostril candidates andnegative nostril candidates.
 13. The method of claim 12 wherein themachine learning boosting algorithm is a Gentle Boost algorithm.
 14. Themethod of claim 8 wherein comparing each nostril candidate matrix seeksto minimize the weighted error with the nostril classifier matrix.
 15. Amobile device comprising: a forward facing camera; a display; a memoryfor storing instructions; and one or more processors configured toexecute the instructions to: obtain a calibration image from thefront-facing camera; locate nostrils in the calibration image; obtain aplurality of tracking images obtained from the front-facing camera andlocate the nostrils in the tracking images; measure a nostrildisplacement of the nostrils between the tracking images and thecalibration image; correlate the nostril displacement to a cervicalspine flexion angle; and compare the cervical spine flexion angle to anacceptable range of cervical spine flexion.
 16. The mobile device ofclaim 15 further comprising an alarm device, and the one or moreprocessors are further configured to associate the difference betweencervical spine flexion angle and the acceptable range of cervical spineflexion to an alarm severity; and indicate the alarm severity to thealarm device.
 17. The mobile device of claim 16 wherein the alarm deviceis a row of lights.
 18. The mobile device of claim 16 wherein the alarmdevice is a progressive scale to indicate the severity of deviation ofthe cervical spine flexion angle from the acceptable range of cervicalspine flexion.
 19. The mobile device of claim 16 wherein the alarmdevice is integrated with the display.
 20. The mobile device of claim 15wherein the one or more processors are further configured to: identifyone or more nostril candidates in the image, the nostril candidatehaving a small pixel window relative to the image size; apply an edgedetection algorithm on the nostril candidates to obtain a nostrilcandidate matrix for each nostril candidate; and compare each nostrilcandidate matrix to a nostril classifier matrix to identify the nostril.